Inter-Rater Reliability: Dependency on Trait Prevalence and Marginal Homogeneity

Researchers have criticized chance-corrected agreement statistics, particularly the Kappa statistic, as being very sensitive to raters’ classification probabilities (marginal probabilities) and to trait prevalence in the subject population. Consequently, several authors have suggested that marginal probabilities be tested for homogeneity and that any comparison between reliability studies be preceded by an assessment of trait prevalence among subjects. The objective of this paper is threefold: (i) to demonstrate that marginal homogeneity testing does not prevent the unpredictable results often obtained with some of the most popular agreement statistics, (ii) to present a simple and reliable inter-rater agreement statistic, and (iii) to gain further insight into the dependency of agreement statistics upon trait prevalence.